from sklearn_benchmarks.reporting.hp_match import HPMatchReporting

Benchmarking prediction: scikit-learn vs. ONNX Runtime

reporting = HPMatchReporting("onnx", config="config.yml")
reporting.run()

KNeighborsClassifier_brute_force

onnx (1.9.0) vs. scikit-learn (1.0.dev0)

All estimators share the following hyperparameters: algorithm=brute.

	estimator	function	n_samples_train	n_samples	n_features	mean_duration	std_duration	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	predict	100000	1000	100	17.962	0.007	0.0	0.018	-1	1	0.662	1.975	0.053	9.095	0.242	See	See
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	17.962	0.007	0.0	0.018	-1	1	0.662	0.027	0.002	676.647	46.747	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.320	0.012	0.0	0.320	-1	1	1.000	1.975	0.053	0.162	0.007	See	See
3	KNeighborsClassifier_brute_force	predict	100000	1	100	0.320	0.012	0.0	0.320	-1	1	1.000	0.027	0.002	12.063	0.949	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	17.757	0.185	0.0	0.018	-1	5	0.700	2.863	0.083	6.202	0.191	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1000	100	17.757	0.185	0.0	0.018	-1	5	0.700	0.028	0.003	623.470	66.575	See	See
6	KNeighborsClassifier_brute_force	predict	100000	1	100	0.325	0.006	0.0	0.325	-1	5	1.000	2.863	0.083	0.113	0.004	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1	100	0.325	0.006	0.0	0.325	-1	5	1.000	0.028	0.003	11.400	1.229	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1000	100	17.767	0.423	0.0	0.018	1	100	0.823	2.303	0.034	7.715	0.216	See	See
9	KNeighborsClassifier_brute_force	predict	100000	1000	100	17.767	0.423	0.0	0.018	1	100	0.823	0.023	0.001	785.889	30.822	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1	100	0.320	0.011	0.0	0.320	1	100	1.000	2.303	0.034	0.139	0.005	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.320	0.011	0.0	0.320	1	100	1.000	0.023	0.001	14.140	0.660	See	See
12	KNeighborsClassifier_brute_force	predict	100000	1000	100	17.526	0.210	0.0	0.018	-1	100	0.823	2.928	0.049	5.986	0.123	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	17.526	0.210	0.0	0.018	-1	100	0.823	0.027	0.004	653.642	100.101	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.313	0.005	0.0	0.313	-1	100	1.000	2.928	0.049	0.107	0.003	See	See
15	KNeighborsClassifier_brute_force	predict	100000	1	100	0.313	0.005	0.0	0.313	-1	100	1.000	0.027	0.004	11.687	1.795	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	17.396	0.135	0.0	0.017	1	5	0.700	2.297	0.021	7.573	0.091	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1000	100	17.396	0.135	0.0	0.017	1	5	0.700	0.024	0.001	734.433	21.366	See	See
18	KNeighborsClassifier_brute_force	predict	100000	1	100	0.333	0.015	0.0	0.333	1	5	1.000	2.297	0.021	0.145	0.007	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1	100	0.333	0.015	0.0	0.333	1	5	1.000	0.024	0.001	14.060	0.738	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1000	100	17.413	0.008	0.0	0.017	1	1	0.662	1.344	0.022	12.953	0.212	See	See
21	KNeighborsClassifier_brute_force	predict	100000	1000	100	17.413	0.008	0.0	0.017	1	1	0.662	0.023	0.001	759.599	23.618	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1	100	0.323	0.019	0.0	0.323	1	1	1.000	1.344	0.022	0.240	0.015	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	100	0.323	0.019	0.0	0.323	1	1	1.000	0.023	0.001	14.085	0.941	See	See
24	KNeighborsClassifier_brute_force	predict	100000	1000	2	4.118	0.035	0.0	0.004	-1	1	0.845	1.634	0.015	2.521	0.032	See	See
25	KNeighborsClassifier_brute_force	predict	100000	1000	2	4.118	0.035	0.0	0.004	-1	1	0.845	0.009	0.004	453.779	200.786	See	See
26	KNeighborsClassifier_brute_force	predict	100000	1	2	0.256	0.009	0.0	0.256	-1	1	0.000	1.634	0.015	0.157	0.006	See	See
27	KNeighborsClassifier_brute_force	predict	100000	1	2	0.256	0.009	0.0	0.256	-1	1	0.000	0.009	0.004	28.254	12.536	See	See
28	KNeighborsClassifier_brute_force	predict	100000	1000	2	4.081	0.061	0.0	0.004	-1	5	0.886	2.430	0.033	1.679	0.034	See	See
29	KNeighborsClassifier_brute_force	predict	100000	1000	2	4.081	0.061	0.0	0.004	-1	5	0.886	0.009	0.003	463.580	173.313	See	See
30	KNeighborsClassifier_brute_force	predict	100000	1	2	0.259	0.010	0.0	0.259	-1	5	0.000	2.430	0.033	0.107	0.004	See	See
31	KNeighborsClassifier_brute_force	predict	100000	1	2	0.259	0.010	0.0	0.259	-1	5	0.000	0.009	0.003	29.460	11.065	See	See
32	KNeighborsClassifier_brute_force	predict	100000	1000	2	4.030	0.036	0.0	0.004	1	100	0.908	1.908	0.026	2.112	0.034	See	See
33	KNeighborsClassifier_brute_force	predict	100000	1000	2	4.030	0.036	0.0	0.004	1	100	0.908	0.002	0.000	1634.499	121.604	See	See
34	KNeighborsClassifier_brute_force	predict	100000	1	2	0.260	0.005	0.0	0.260	1	100	0.000	1.908	0.026	0.136	0.003	See	See
35	KNeighborsClassifier_brute_force	predict	100000	1	2	0.260	0.005	0.0	0.260	1	100	0.000	0.002	0.000	105.441	8.001	See	See
36	KNeighborsClassifier_brute_force	predict	100000	1000	2	4.071	0.047	0.0	0.004	-1	100	0.908	2.432	0.043	1.674	0.035	See	See
37	KNeighborsClassifier_brute_force	predict	100000	1000	2	4.071	0.047	0.0	0.004	-1	100	0.908	0.008	0.002	484.312	106.208	See	See
38	KNeighborsClassifier_brute_force	predict	100000	1	2	0.277	0.010	0.0	0.277	-1	100	0.000	2.432	0.043	0.114	0.004	See	See
39	KNeighborsClassifier_brute_force	predict	100000	1	2	0.277	0.010	0.0	0.277	-1	100	0.000	0.008	0.002	32.950	7.309	See	See
40	KNeighborsClassifier_brute_force	predict	100000	1000	2	4.136	0.072	0.0	0.004	1	5	0.886	1.865	0.026	2.218	0.050	See	See
41	KNeighborsClassifier_brute_force	predict	100000	1000	2	4.136	0.072	0.0	0.004	1	5	0.886	0.002	0.000	1681.971	135.955	See	See
42	KNeighborsClassifier_brute_force	predict	100000	1	2	0.254	0.004	0.0	0.254	1	5	0.000	1.865	0.026	0.136	0.003	See	See
43	KNeighborsClassifier_brute_force	predict	100000	1	2	0.254	0.004	0.0	0.254	1	5	0.000	0.002	0.000	103.166	8.314	See	See
44	KNeighborsClassifier_brute_force	predict	100000	1000	2	4.119	0.050	0.0	0.004	1	1	0.845	1.139	0.022	3.618	0.083	See	See
45	KNeighborsClassifier_brute_force	predict	100000	1000	2	4.119	0.050	0.0	0.004	1	1	0.845	0.002	0.000	2171.382	274.745	See	See
46	KNeighborsClassifier_brute_force	predict	100000	1	2	0.263	0.012	0.0	0.263	1	1	0.000	1.139	0.022	0.231	0.011	See	See
47	KNeighborsClassifier_brute_force	predict	100000	1	2	0.263	0.012	0.0	0.263	1	1	0.000	0.002	0.000	138.685	18.555	See	See

KNeighborsClassifier_kd_tree

onnx (1.9.0) vs. scikit-learn (1.0.dev0)

All estimators share the following hyperparameters: algorithm=kd_tree.

	estimator	function	n_samples_train	n_samples	n_features	mean_duration	std_duration	iteration_throughput	latency	n_jobs	n_neighbors	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	141.278	0.000	0.0	0.141	1	100	0.981	5.702	0.046	24.777	0.201	See	See
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	141.278	0.000	0.0	0.141	1	100	0.981	0.005	0.001	28981.818	6339.436	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.778	0.152	0.0	2.778	1	100	1.000	5.702	0.046	0.487	0.027	See	See
3	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.778	0.152	0.0	2.778	1	100	1.000	0.005	0.001	569.881	128.504	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	138.909	0.000	0.0	0.139	-1	1	0.963	0.538	0.017	258.206	8.287	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	138.909	0.000	0.0	0.139	-1	1	0.963	0.004	0.001	36286.767	9096.659	See	See
6	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.774	0.106	0.0	2.774	-1	1	1.000	0.538	0.017	5.156	0.257	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.774	0.106	0.0	2.774	-1	1	1.000	0.004	0.001	724.609	183.744	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	150.134	0.000	0.0	0.150	1	5	0.982	1.696	0.022	88.513	1.158	See	See
9	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	150.134	0.000	0.0	0.150	1	5	0.982	0.002	0.001	66876.656	18357.060	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.735	0.056	0.0	2.735	1	5	1.000	1.696	0.022	1.612	0.039	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.735	0.056	0.0	2.735	1	5	1.000	0.002	0.001	1218.141	335.311	See	See
12	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	149.447	0.000	0.0	0.149	1	1	0.963	0.883	0.015	169.161	2.863	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	149.447	0.000	0.0	0.149	1	1	0.963	0.002	0.000	82931.865	16902.316	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.918	0.089	0.0	2.918	1	1	1.000	0.883	0.015	3.303	0.116	See	See
15	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.918	0.089	0.0	2.918	1	1	1.000	0.002	0.000	1619.271	333.738	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	154.124	0.000	0.0	0.154	-1	5	0.982	1.047	0.028	147.218	3.984	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	154.124	0.000	0.0	0.154	-1	5	0.982	0.004	0.001	35515.279	6610.423	See	See
18	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.804	0.156	0.0	2.804	-1	5	1.000	1.047	0.028	2.679	0.166	See	See
19	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.804	0.156	0.0	2.804	-1	5	1.000	0.004	0.001	646.209	125.561	See	See
20	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	159.426	0.000	0.0	0.159	-1	100	0.981	3.376	0.048	47.224	0.665	See	See
21	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	159.426	0.000	0.0	0.159	-1	100	0.981	0.007	0.001	21972.877	4496.817	See	See
22	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.798	0.138	0.0	2.798	-1	100	1.000	3.376	0.048	0.829	0.042	See	See
23	KNeighborsClassifier_kd_tree	predict	1000000	1	10	2.798	0.138	0.0	2.798	-1	100	1.000	0.007	0.001	385.685	81.186	See	See
24	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.063	0.002	0.0	0.000	1	100	0.890	0.041	0.001	1.546	0.068	See	See
25	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.063	0.002	0.0	0.000	1	100	0.890	0.001	0.000	78.264	13.594	See	See
26	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.006	0.000	0.0	0.006	1	100	0.000	0.041	0.001	0.145	0.012	See	See
27	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.006	0.000	0.0	0.006	1	100	0.000	0.001	0.000	7.354	1.375	See	See
28	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.040	0.002	0.0	0.000	-1	1	0.898	0.032	0.004	1.248	0.176	See	See
29	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.040	0.002	0.0	0.000	-1	1	0.898	0.003	0.001	15.046	3.026	See	See
30	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.006	0.001	0.0	0.006	-1	1	1.000	0.032	0.004	0.178	0.035	See	See
31	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.006	0.001	0.0	0.006	-1	1	1.000	0.003	0.001	2.142	0.519	See	See
32	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.041	0.001	0.0	0.000	1	5	0.899	0.033	0.004	1.265	0.146	See	See
33	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.041	0.001	0.0	0.000	1	5	0.899	0.001	0.000	53.493	14.227	See	See
34	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.006	0.000	0.0	0.006	1	5	1.000	0.033	0.004	0.178	0.025	See	See
35	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.006	0.000	0.0	0.006	1	5	1.000	0.001	0.000	7.508	2.083	See	See
36	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.041	0.001	0.0	0.000	1	1	0.898	0.027	0.001	1.528	0.071	See	See
37	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.041	0.001	0.0	0.000	1	1	0.898	0.001	0.000	52.235	11.126	See	See
38	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	1	1	1.000	0.027	0.001	0.202	0.008	See	See
39	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.005	0.000	0.0	0.005	1	1	1.000	0.001	0.000	6.922	1.466	See	See
40	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.041	0.002	0.0	0.000	-1	5	0.899	0.032	0.003	1.281	0.126	See	See
41	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.041	0.002	0.0	0.000	-1	5	0.899	0.002	0.000	17.033	2.239	See	See
42	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.006	0.000	0.0	0.006	-1	5	1.000	0.032	0.003	0.170	0.018	See	See
43	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.006	0.000	0.0	0.006	-1	5	1.000	0.002	0.000	2.263	0.314	See	See
44	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.062	0.001	0.0	0.000	-1	100	0.890	0.045	0.005	1.377	0.170	See	See
45	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.062	0.001	0.0	0.000	-1	100	0.890	0.003	0.000	24.262	4.608	See	See
46	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.007	0.001	0.0	0.007	-1	100	0.000	0.045	0.005	0.152	0.032	See	See
47	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.007	0.001	0.0	0.007	-1	100	0.000	0.003	0.000	2.672	0.686	See	See

HistGradientBoostingClassifier_best

onnx (1.9.0) vs. scikit-learn (1.0.dev0)

All estimators share the following hyperparameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

	estimator	function	n_samples_train	n_samples	n_features	n_iter	mean_duration	std_duration	iteration_throughput	latency	accuracy_score	mean_duration_onnx	std_duration_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	predict	100000	1000	100	300	0.482	0.028	0.002	0.0	0.815	0.159	0.006	3.023	0.206	See	See